System with fabric modules

ABSTRACT

In some examples, a chassis contains a fabric module and a plurality node modules that are arranged in a plurality of rows. The fabric module is positioned in a space between a first row and a second row of the plurality of rows, and the fabric module is connected to at least two node modules of the plurality of node modules to provide communications connectivity between the at least two node modules, the chassis to accept longitudinal insertion in a longitudinal direction of the plurality of node modules and the fabric module, the fabric module being removable in the longitudinal direction from the chassis by moving the fabric module in the space between the first row and the second row without first removing the node modules in the plurality of rows.

CROSS REFERENCE TO RELATED APPLICATIONS

This is a continuation of U.S. application Ser. No. 13/808,507, filedJan. 4, 2013, which is a national stage application under 35 U.S.C. §371of PCT/US2010/048970, filed Sep. 15, 2010, both hereby incorporated byreference.

BACKGROUND

A typical blade system includes a chassis for holding several “blades”.Each blade can include one or more processor nodes, each of whichincludes one or more processors and associated memory. The chassis caninclude a backplane that provides power and connectivity, input/output(I/O) connectivity including network connectivity and inter-bladeconnectivity. In some blade systems, front connector bars spanning twoor more blades provide or supplement inter-blade connectivity. In someblade systems, the inter-blade connectivity provides for cache coherentoperation among processor blades associated with different blades. Thisallows a set of blades to operate as a single more powerful computerrather than as a network of separate computers that happen to be in thesame chassis. Blade systems can be upgraded conveniently by swappingprevious-generation blades with more capable newer-generation blades. Inthis sense, blade systems provide a hedge against obsolescence and allowa customer's investment to be amortized over a longer time, decreasingthe overall cost of ownership.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a modular computer system in accordancewith an embodiment.

FIG. 2 is a schematic diagram of a fabric module of the computer systemof FIG. 1.

FIG. 3 is a flow chart of a process implementable in the computer systemof FIG. 1.

FIG. 4 is a flow diagram for a module swap operation and showing frontand side views of a blade computer system in accordance with anembodiment.

FIG. 5 is a front schematic view of a chassis of the computer system ofFIG. 4.

FIG. 6 is a schematic view of a blade of the computer system of FIG. 4.

FIG. 7 is a schematic view of a fabric module of the computer system ofFIG. 4.

FIG. 8 is a schematic view of a fabric module of the computer system ofFIG. 4.

FIG. 9 is a schematic view of a fabric module of the computer system ofFIG. 4.

FIG. 10 is a flow chart of a process implementable in the system of FIG.4.

FIG. 11 is a schematic diagram for a fabric module installable in thesystem of FIG. 4.

FIG. 12 is a schematic diagram for a fabric module installable in thesystem of FIG. 4.

DETAILED DESCRIPTION

A modular computer system 100, shown in FIG. 1, includes a chassis 101,node modules N1 and N2, and a fabric module 107 that provides routing109 over which node modules N1 and N2 communicate with each other. Tothis end, fabric module 107 includes connectors C1 and C2 on its topface 109 for connecting to respective node modules, e.g., node modulesN1 and N2, as shown in FIG. 2. Fabric module 107 can be inserted orremoved at a process segment 301, in the same longitudinal dimension(into or out of the page given the front view of FIG. 1.) that nodemodules N1 and N2 can be removed. Node modules N1 and N2 can be runcooperatively, e.g., as a single computer, with fabric module 107installed at a process segment 302 (which can occur before a removal orafter an insertion of process segment 301).

Herein, “module” refers to a hardware entity that can be inserted into achassis. The terms “node module” and “fabric module” are definedrelative to each other so that a “fabric module” is a module configuredto provide communications connectivity between or among node modules.The node modules can include general-purpose or application-specificcomputer modules (providing processing, storage, and communications,e.g., I/O and networking”, modules that emphasize one data handlingfunction, e.g., storage modules, connectivity modules (e.g., networkswitches dedicated to one or more network layers), application-specifichardware (e.g., sensors and controllers). A “processor module” is acomputer module including one or more processor nodes (each of which caninclude one or more processors and memory). “Blade” herein, refers to atype of node module having a physically “thin” configuration.

Unlike systems in which connectivity is provided by a backplane or amidplane, system 100 permits connectivity to be upgraded withoutreplacing an entire chassis. Fabrics typically can accommodate a smallnumber of computer (e.g., processor) upgrades, but eventually areoutstripped by the capabilities of the newer computers. Usingreplaceable fabric modules allows the chassis to be retained, e.g., fora decade or more rather than for just a few years, through moregenerations of upgrades. Furthermore, fabric module 107 does not impedefront-to-back airflow. Relative to systems that provide inter-bladeconnectivity using front connector bars, serviceability is improved asfabric module 107 does not have to be removed to replace a connectednode module.

A blade system 400 includes a chassis 401, a vertically adjacent pair ofrows of blades (node modules) B10 and B20, and fabric modules FM1, FM2,and FM3, as shown in FIG. 4. Row B10 includes blades B11, B12, B13, andB14. Row B20 includes blades B21, B22, B23, and B24. Other embodimentsprovide for different numbers of rows and different numbers of bladesper row. Also, alternative embodiments provide for different numbers offabric modules for a given number of blade rows. For example, top andbottom fabric modules, such as FM1 and FM3 in FIG. 4, can be omitted.

As shown in the bottom half of FIG. 4 for blade B24 and fabric moduleFM2, blades B11-B14, blades B21-B24, and fabric modules FM1-FM3 are allremovable and insertable through the front 403 of chassis 401 usinglongitudinal motions. Herein, a horizontal-vertical-longitudinalcoordinate system is used. “Horizontal” is the dimension in which theblades of a row are spaced; for example, blades B11-B14 are arrangedleft to right along the horizontal dimension. “Vertical” is thedimension in which a fabric module is spaced from a row of bladesconnected to it; for example, fabric module FM1, row B10, fabric moduleFM2, row B20, and fabric module FM3 are arranged bottom to top along thevertical dimension. “Longitudinal” refers to the dimension of insertionfor modules. In the illustrated embodiment, these dimensions aresubstantially orthogonal to each other, in other words, each pair ofdimensions defines an acute or right angle more than 45° so that theyare more orthogonal than aligned. Herein, terms such as “front”, “rear”,“top”, “bottom”, “left”, “right” “above”, and “below” are to beinterpreted in the context of the coordinate system.

Chassis 401, shown separately in FIG. 5, provides blade (node module)slots 501 and fabric-module slots 503 for receiving, guiding, andsecuring blades and fabric modules. In some embodiments, other nodemodules, e.g., network modules, can be installed in the vertical bladeslots, with computer-network linkages through fabric modules. Inaddition, chassis 401 provides power connections 505 for all modules(including blades) and data connections e.g., with peripherals andnetworks for all blades. Other embodiments include cam and clampingmechanisms for securing modules and for effecting connections betweenblades and fabric modules. Also, in some embodiments, chasses includeintegrated cooling, e.g., fans or liquid cooling features. Also, in someembodiments, management features are provided, e.g., virtual ports.

Blades B12-B14 and B21-B24 are similar to B11, described with referenceto FIG. 6. Blade B11 is a processor module including two processor nodesP1 and P2. In alternative embodiments, blades can include one processornode or more than two processor nodes; different blades can havedifferent numbers of processors and amounts of memory per node.Processor node N1 includes four processors CP1, memory (RAM) ME1, andstorage (ST1). Processor node N2 includes four processors CP2, memory(RAM) ME2, and storage (ST2). Processor nodes P1 and P2 are coupled sothat they can operate coherently—i.e., as though memories ME1 and ME2constituted a unified memory that can be addressed directly by anyprocessor of nodes N1 and N2.

Blade B11 includes a top connector 611 and a bottom connector 613 forconnecting to vertically adjacent fabric modules, e.g., modules FM2 andFM1, respectively in FIG. 4. Each blade can be configured so that zero,one, or both of its top and bottom connectors are active. Blade B11 alsoincludes on its rear face 635 power and network connectors 621 and 623,respectively, for receiving power and establishing specializedconnectivity, e.g., such as for management. Connectors 621 and 623 areon the top 631 and bottom 633 of blade B11 respectively, more toward therear face 635 of blade B11 than its front face 637.

Various embodiments employ different techniques to achieve connectivitywith communication signals: (a) card edge connector, where the card edgeof the blade module slides between two adjacent connectors, (b) cam downor up actuation, where the blade slides in and is pressed down against atraditional connector on the shelf via a cam action, (c) flex-cableconnector, where the blade slides in and the connector's flex cableextension is pressed down or up against a shelf using a cam action, and(d) optically interconnected systems where the shelf contains optical“traces” (e.g., waveguides) and the connection may be made with aslide-by, self-aligning set of optical connectors.

Fabric module FM2 is represented in FIG. 7. Routing glue logic 701provides a switched star topology for full 8×8 routing. (Connectorsshown using dashed lines are on the bottom of fabric module FM2). Aparticular routing can be selected via a front panel of fabric moduleFM2 or by a management station connected to blade system 400. Inaddition to routing, routing glue logic 701 provides other gluefunctions such as signal buffering and snoop filtering.

Fabric module FM1 is represented in FIG. 8. It is adapted to couple thetwo rightmost blades and the two leftmost blades in a row. Fabric moduleFM3 is depicted in FIG. 9. It is adapted to connect four blades in arow. In an alternative embodiment, a fabric module has the pattern andconnectors shown in FIG. 8 on its top face and the pattern andconnectors of FIG. 9 on its bottom face; other alternative embodimentscan have different patterns on their top and bottom faces. In some suchembodiments, a connection configuration can be changed by inverting afabric module with different patterns (and connectors) on its top andbottom faces even though the fabric module itself is not dynamicallyreconfigurable.

In practice, computer system 400, FIG. 4, could be operated with theblades communicatively connected to fabric module FM2 and notcommunicatively coupled to fabric modules FM1 and FM3, e.g., at aprocess segment 1001 of a process 1000 flow charted in FIG. 10. Toprepare for repairing or replacing fabric module FM2, process segment1002 provides for breaking the connection between fabric module FM2 andany blades connected to it. If it is desired to run some of these bladescoherently, they can be reconnected at process segment 1003 using otherfabric modules, e.g., fabric modules FM1 and FM3. This may not result inthe exact configuration provided by fabric module FM2, but may stillimprove upon running blades individually.

Fabric module FM2 can be removed longitudinally at process segment 1004.The removed fabric module can be repaired, inverted, or replaced atprocess segment 1005. Once repair is complete, the module inverted, or areplacement is found, the resulting fabric module can be insertedlongitudinally at process segment 1006. To effect coherent processingvia connections of the newly inserted fabric modules, Communicativeconnections to the other fabric modules can be broken and communicativeconnections to the newly inserted module established at process segment1007.

A couple of other fabric modules FM4 and FM5 are shown in FIGS. 11 and12, respectively. Fabric module FM4 provides four-point connectionsbetween vertical pairs of two-node blades. Thus, for example, the twoprocessor nodes of blade B11 and the two processor nodes of blade B21are grouped to form a four processor-node set that runs coherently.Fabric module FM5 provides eight-point connections for left and rightblocks of four blades.

Blade system 400 provides for fabric modules of many other glued andglueless configurations. Also, communicative connections can beelectrical, optical or both. Some fabric modules may provide fordifferent connection configurations by inverting the fabric modulevertically, i.e., installing it after a 180° rotation about alongitudinal axis. Some fabric modules may have fixed configurations;others may be reconfigurable while installed. Connections can be madeusing more than one fabric module by using both top and bottom bladeconnectors. Such connections can be made through the processor nodes ofa blade. Also, a specially designed blade can allow a direct connectionbetween upper and lower fabric modules. For example, blade B21 of FIG. 4provides for a direct connection between fabric modules FM2 and FM3.

System 400 thus provides for a customer installable fabric module“shelf”; in general, this can mean that the cost for high speedblade-to-blade connectivity is only paid when it is used. Connectedblades can be independently serviced or upgraded. One chassis can beenabled for a variety of configurations depending on the nature of thefabric modules installed. Also, in its horizontal orientation, a fabricmodule does not impede front-to-back airflow, unlike a traditionalfront-plane or mid-plane. Furthermore, the interconnect shelf itself canbe independently serviced or upgraded.

In other blade systems, blades may be swappable through the front faceof a chassis while fabric modules are swappable through the rear. Thisconfiguration can allow connections between fabric and blades that areeffected by abutting instead of sliding by each other. Some bladesystems use one-sided patterns; to connect to blade both above andbelow, a pair of fabric modules can be inserted back-to-back. Theback-to-back connection can provide inter-layer connections between thefabric modules and thus between rows of blades. Some blade systems useboth horizontal and vertical fabric modules (and corresponding slots),defining a rectangular array of “cubbie holes”, each of which can holdmultiple compute and network blades.

In some embodiments, a chassis configured to hold processor modules andone or more fabric modules is unpopulated, i.e., does not presently holdany processor modules or fabric modules. In other embodiments, such achassis is partially or completely populated by other types of modules,e.g., modules having primary purposes, e.g., data input or output orboth, other than data processing or inter-module routing. “Coherently”herein applies to a multi-processor system for which multiple processorstreat a common or collective memory as a unified whole having a singleconsistent state. “Cooperatively” means working interactively (eithercoherently or non-coherently) to accomplish a goal.

Herein, a “system” is a set of interacting elements, wherein theelements can be, by way of example and not of limitation, mechanicalcomponents, electrical elements, atoms, the physically encoded forms ofinstructions encoded in storage media, and process segments. In thisspecification, related art is discussed for expository purposes. Relatedart labeled “prior art”, if any, is admitted prior art. Related art notlabeled “prior art” is not admitted prior art. The illustrated and otherdescribed embodiments, as well as modifications thereto and variationsthereupon are within the scope of the following claims.

What is claimed is:
 1. A system comprising: a chassis; a fabric modulecontained inside the chassis; and a plurality node modules containedinside the chassis and arranged in a plurality of rows, the fabricmodule positioned in a space between a first row and a second row of theplurality of rows, the fabric module physically connected to at leasttwo node modules of the plurality of node modules to providecommunications connectivity between the at least two node modules, thechassis to accept longitudinal insertion in a longitudinal direction ofthe plurality of node modules and the fabric module, the fabric modulebeing removable in the longitudinal direction from the chassis by movingthe fabric module in the space between the first row and the second rowwithout first removing the node modules in the plurality of rows.
 2. Thesystem of claim 1, wherein the chassis is to provide power to theplurality of node modules when held by the chassis.
 3. The system ofclaim 1, wherein the node modules include processor modules, and thefabric module provides for coherent processing between at least two ofthe processor modules.
 4. The system of claim 1, wherein the chassis isto hold vertically adjacent pairs of horizontal rows of node modules andplural fabric modules.
 5. The system of claim 1, wherein the at leasttwo node modules comprise processor modules, and the communicationsconnectivity provided by the fabric module between the processor modulesprovide for coherent processing between the processor modules.
 6. Thesystem of claim 5, wherein the fabric module is to provide for differentgroupings of coherent processor modules in response to commands withoutbeing removed from the chassis.
 7. The system of claim 5, wherein thefabric module is to provide for glueless coherent processing betweenprocessor modules of different rows of the plurality of rows.
 8. Thesystem of claim 1, wherein the fabric module is to provide for differentgroupings of coherent processor modules according to a physicalinversion state of the fabric module.
 9. The system of claim 1, whereinthe fabric module is a first fabric module, the system furthercomprising a second fabric module, and wherein after the first fabricmodule is removed from the chassis and disconnected from the at leasttwo node modules, the at least two node modules are communicativelyconnected to the second fabric module to establish communicationsconnectivity with the second fabric module.
 10. A method comprising:providing, in a chassis, a fabric module; providing, in the chassis, aplurality node modules arranged in a plurality of rows; positioning thefabric module in a space between a first row and a second row of theplurality of rows; and connecting the fabric module to at least two nodemodules of the plurality of node modules to provide communicationsconnectivity between the at least two node modules, the chassis toaccept longitudinal insertion in a longitudinal direction of theplurality of node modules and the fabric module, the fabric module beingremovable in the longitudinal direction from the chassis by moving thefabric module in the space between the first row and the second rowwithout first removing the node modules in the plurality of rows. 11.The method of claim 10, wherein the first and second rows are verticallyadjacent rows of node modules.
 12. The method of claim 10, furthercomprising, while the fabric module is in the space, reconfiguring thefabric module so as to change a grouping of node modules.
 13. The methodof claim 10, further comprising communicatively coupling the fabricmodule and another fabric module through one of the plurality of nodemodules disposed between the fabric modules.
 14. The method of claim 10,further comprising after removing a first fabric module of the fabricmodules, reinserting, into the chassis, the first fabric module invertedrelative to its previous orientation.
 15. The method of claim 9, whereinat least some of the node modules are processor modules, and the fabricmodule provides for glueless coherent running of a group of theprocessor modules.